DOCUMENT RESUME 



ED 480 927 



HE 036 152 



AUTHOR 

TITLE 

INSTITUTION 
REPORT NO 
PUB DATE 
NOTE 
PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Belcheir, Marcia J. 

Impact of the Change in English Cutscores on Grades. Research Report 
2002-04. 

Boise State Univ., ID. Office of Institutional Assessment. 
BSU-RR-2002-04 
2002-06-00 

lip.; Colored figures may not reproduce well. 

Reports - Evaluative (142) 

EDRS Price MFOl/PCOl Plus Postage. 

College English; College Entrance Examinations; ^College Freshmen; 
^Cutting Scores; ^English Curriculum; Higher Education; ^Student 
Placement 

*Boise State University ID 



ABSTRACT 

The grades of students in English classes at Boise State University, 
Idaho, were studied before and after the implementation of changes in the cutscores 
for placing entering students in three different English levels. The effects of the 
cut score changes were to move more students into English 090 (the lowest level) from 
English 101, and to move more students from English 101 to English 102 (more 
advanced) . The change also allowed some high-scoring students to receive credit for 
English 102 when they would previously have had to take the course. Information was 
provided on the course grades of students who were affected by the changes, compared 
to those who were not, and course grade distributions before and after the changes 
were available. Findings seem to indicate a lack of relationship between scores on the 
tests used for placement, the ACT and the Scholastic Assessment Tests. This finding 
may not be surprising, since these tests were not developed for course placement 
purposes. Grades distributions for the three courses were similar before and after the 
change in cutscores. Grades improved in English 101 after more students were required 
to take English 090. Although assigning more students to English 090 slowed their 
academic progress, these students probably were better equipped to take the next 
level, English 101. (SLD) 
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This study reports on the grades of students in English classes 
prior to and following the implementation of changes in the 
cutscores for placing students in ENGL 090, 101, or 102. The 
effects of the cutscore changes were to (1) move more students into 
ENGL 090 from ENGL 101, (2) move more students into ENGL 
^ 102 from ENGL 101, and (3) give credit for ENGL 102 to some 
high-scoring students who previously would have had to take the 
course. Information was provided on course grades of students who 
were affected by the changes compared to those who were not, course 
jrade distributions before and after the cutscore changes, and grades in 
le subsequent course before and after the cutscore changes. 

:fore the change, students who were slated to be moved into ENGL 090 
about the same percentage of “D,” “F,” and “W” grades as those who 
mined in ENGL 101. After the change, similar percentages of 
issigned” students passed ENGL 090 compared to their lower-scoring 
interparts who had always needed to take 090. A similar pattern of “no 
jrences” was found for students who were moved into ENGL 102. This 
ing seems to indicate a lack of relationship between test scores and 
rse grades. Perhaps this is not surprising since neither the ACT or SAT 
e developed for course placement purposes. 

Grade distributions in ENGL 090, 101, and 102 also were similar both 
before and after the change in cutscores. However, grades improved in 
ENGL 101 after more students were required to take ENGL 090. Thus, 
although assigning more students to ENGL 090 slowed their academic 
^ progress, these students were probably better equipped to take the next 
^ level of English, ENGL 101. 
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Beginning with the Fall of 2000, the State Board of Education standardized the scores across 
public institutions of higher education for the placement of students in English and Mathematics 
courses. The change came about because of discrepancies between institutions within Idaho on 
the cutscores. Students with the same score, for example, might be assigned to ENGL 101 at one 
institution and to ENGL 102 at another institution. At the urging of the Board, representatives 
from Idaho’s public institutions met to agree on the common cutscores, with the result that some 
of Boise State’s cutscores were modified. Some students who previously would have enrolled in 
ENGL 101 were reassigned to ENGL 090, while others were moved in ENGL 102. Students 
with especially high scores were simply given credit for ENGL 102 under the new system. The 
prior cutscores which Boise State used for placement in English and the current statewide 
cutscores are displayed in Appendix A at the end of this report. 

In the aftermath of these changes, the English department noted some differences in student 
performance and wished to have empirical data to see if this type of data corroborated their more 
informal observations. The Office of Institutional Assessment worked with the department and 
developed the following key questions to be answered from the data: 

• Did students who had scores which fell in the area of cutscore changes perform 
differently than students who were not affected? 

• Have grade distributions for first-time enrollees in ENGL 090, 101, and 102 changed 
since the cutscore changes? 

• Did the cutscore changes appear to have an affect on performance in the next English 
course? 

To answer these questions, test scores and grades were compiled from students who enrolled in 
ENGL 090, 101, or 102 for the first time between Fall 1998 and Fall 2001. The “pre-cutscore 
change” group included students who enrolled in English for the first time in Fall 1998, Spring 
1999, Fall 1999, or Spring 2000. The “post-cutscore change” group included students who 
enrolled for the first time in Fall 2000, Spring 2001, or Fall 2001. Summer school enrollment 
was not included in the analysis. 

ResuIts 



Did students who had scores which fell in the area of cutscore changes perform differently than 
students who were not affected? 



In an ideal world, cutscores are modified because they are not working for students in a 
particular score range. Perhaps students with scores close to the cutscore are performing so well 
that they could be moved to the next level of class (e.g., they could be moved from ENGL 101 to 
ENGL 102). Conversely, they may be at the bottom on the cutscore range and struggling, 
indicating that they need to start at a lower level (e.g., they could be moved from ENGL 101 to 
ENGL 090). With the implementation of the statewide set of cutscores, some students who 
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previously would have taken ENGL 101 were assigned to ENGL 090 or ENGL 102. In addition, 
some students who formerly were required to take ENGL 102 were now simply given credit for 
it based on their test scores. 

To see whether these changes were warranted from a measurement perspective, two approaches 
were taken. First, test scores and grades for students who took their first English course prior to 
the cutscore changes were obtained and grades were compared, looking for evidence that the 
affected group’s grades indicated a need for change. Second, test scores and grades for students 
who took their first English course after the cutscore changes (Fall 2000-Fall 2001) were 
obtained. The grades of students who previously were newly assigned to the class with the 
cutscore change were compared to students whose class assignment did not change. 

Moving Students into ENGL 090 

The first change involved shifting the cutscores so that more Boise State students enrolled in 
ENGL 090 who formerly would have taken ENGL 101. Were students with scores in the 
affected range performing so poorly in ENGL 101 that they needed to be moved to ENGL 090? 
How did students in this range fare in ENGL 090 after the change in cutscores? Did their 
performance differ significantly from the performance of students whose scores were low 
enough that they were always assigned to ENGL 090? 

As shown by Figure 1 below, the evidence is weak that the shift into ENGL 090 needed to be 
made for measurement reasons. Less than 20% of the students failed ENGL 101 who had scores 
in the range that would have placed them in ENGL 090 had they enrolled after the cutscore 
change was implemented. Furthermore, their performance was similar to that of students with 
scores which would have allowed them to remain in ENGL 101. 

Figure 1. Comparison of D,F, & W Grades in ENGL 101 Prior to Cutscore Changes 



■S 

I. 

O 

uT 
d 



17.71 



Moved to 090 



16.24 




Remained in 101 



Another way to assess the effect of the cutscore changes is to look at actual performance in the 
classroom after the cutscore change. Did students with scores in the range that moved them into 
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ENGL 090 perform significantly better than those who had scores that always placed them into 
ENGL 090? Figure 2 provides some answers. Note that in three semesters, over 250 students 
took ENGL 090 who previously would have taken ENGL 101, and that a similar percentage 
passed the course compared to students with lower scores. This finding indicates that the 
cutscore change may have slowed student progress because of unnecessary coursework. 

Figure 2. Comparison of Passing Grades in ENGL 090 After Cutscore Change 
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Moving Students into ENGL 102 

A second change involved shifting testscores so that some students enrolled in ENGL 102 who 
previously would have taken ENGL 101 . Were students within this score range doing so well 
that it was advantageous to them to skip ENGL 101? How did they perform once the cutscore 
change was implemented and they enrolled in ENGL 102 instead of 101? How did their 
performance compare to students who were not reassigned by the test scores? 

As shown by Figure 3 below, again there was scant evidence from a measurement perspective 
that a change needed to be made. About two-thirds of the students who would have been moved 
into ENGL 102 with the cutscore change received “As” or “Bs” in ENGL 101 . Their 
performance was essentially the same as that of students who would remain in ENGL 101 
because of their lower test scores. 
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Figure 3. Comparison of A & B Grades in ENGL 101 Prior to Cutscore 

Change 
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Again, performance in classes following the cutscore changes also provided information. About 
120 students were able to skip ENGL 101 in the three semesters following the change. About 
20% of this group received a “D,” “F,” or “W” in the course, which is similar to the performance 
of students with higher scores who remained assigned to ENGL 102 (see Figure 4 below). 

Figure 4. Comparison of D,F,&W Grades in ENGL 102 After Cutscore 

Change 
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Cutscores also were changed so that students who performed very well on the test were allowed 
to skip ENGL 102 altogether and receive credit instead. Did students who took ENGL 102 
before the change perform so well in the course that they should skip it altogether? 

Results indicate that about 85% of the students who received credit would have attained an “A” 
or a “B” in the course (Figure 5). This performance was decidedly better than that of students 
whose scores caused them to remain in ENGL 102. In a two year timeframe, however, only 
about 35 students would be affected by the change (assuming a stable enrollment). 
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Figure 5. Comparison of A & B Grades in ENGL 102 Prior to Cutscore Change 
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Thus, from a measurement perspective, the shifts in cutscores were unnecessary because grades 
were so similar between the groups. By placing more students into ENGL 090, the academic 
progress for about 250 students in three semesters was slowed unnecessarily. However, 
evidence does indicate that students who were given credit for ENGL 102 were top performers. 
This acceleration of academic progress, however, only affected about 35 students in three 
semesters. 

Did the grade distributions change after the cutscore changes were implemented? 

The thinking behind this question was that if the cutscore changes had an impact, in general we 
should see it in the grades given to students. Figure 6 below shows the percentage who passed 
ENGL 090 before and after the cutscore change. Note that slightly fewer passed following the 
change, though we would expect more to pass because students with higher academic abilities 
(as measured by test scores) had been added to the class. The differences were not statistically 
significant. 
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Figure 6. Percent passing ENGL 090 before and after the cutscore 

changes 
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ENGL 101 grade distributions also did not change in any statistically significant fashion. 

As shown by Table 1, only slightly more students received “As” following the change compared 
to before the change in cutscores. The same result was found for ENGL 102, with grade 
distributions remaining much the same before and after the cutscore changes. 

Table 1. ENGL 101 and 102 Grade Distributions Before and After the Cutscore Changes 



Group 


Percent of Grades that were: 


F, W 


D 


C 


B 


A 




ENGL 101 Grades 


Before change (N=4,862) 


12.40 


3.78 


17.52 


33.30 


32.99 


After change (N=3,352) 


12.62 


3.13 


15.81 


33.29 


35.14 




ENGL 102 Grades 


Before change (N=2,300) 


11.87 


3.04 


13.30 


32.52 


39.26 


After change (N=2,021) 


13.81 


3.71 


12.32 


31.07 


39.09 



Thus, grade distributions did not change significantly for any of the three classes. There are a 
number of explanations for this. One is that the tests are not closely related to class performance 
Another is that faculty shifted their expectations to deal with the changes in academic level, the 
result being similar grade distribution. A third possible explanation is that final grade is a 
measure that is too gross to measure subtle changes in classroom performance. These changes, 
however, might still be evident in the next English course. 

Did cutscore changes have an affect on performance in the next English course? 

Even though grade distributions might not show a difference for the first course that students 
took, differences might appear in the next level course. For example, the latest assessment 
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results for ENGL 101 showed a marked improvement in student skills, leading to speculation 
that having more students first take ENGL 090 provided a better background to handle ENGL 
101. This analysis assesses grades for (a) students who began in ENGL 090 before moving to 
ENGL 101 and (b) students who began in ENGL 101 before moving to ENGL 102. 

As shown by Table 2, sending more students to ENGL 090 with the change in cutscores was 
related to in better grades in ENGL 101. After the change, 73% of the students who began in 
ENGL 090 received “A” or “B” grades compared to only 56% prior to the change. It appears, 
therefore, that raising the cutscores was ultimately beneficial to students who began their 
academic career with weak writing skills. 

Table 2. ENGL 101 Grades for Students Whose First Course was ENGL 090 



Group: 


Percent Receiving the Following Grades in ENGL 101 


F, W 


D 


C 


B 


A 


Before change in cutscores 
(N=233) 


12.45 


5.58 


26.18 


34.33 


21.46 


After change in cutscores 
(N=150) 


6.00 


3.33 


18.00 


43.33 


29.33 



No significant change was found, however, for students who began in ENGL 101 before moving 
on to ENGL 102 (see Table3). Recall, however, that the main change to ENGL 101 was to 
remove both high and low scoring students through the cutscore changes. This is evident from 
the reduced number of students enrolled in ENGL 101 after the cutscore changes. 

Table 3. ENGL 102 Grades for Students Whose First Course was ENGL 101 



Group: 


Percent Receiving the Following Grades in ENGL 102 


F, W 


D 


C 


B 


A 


Before change in cutscores 
(N= 1,929) 


11.98 


4.46 


18.51 


32.71 


32.35 


After change in cutscores 
(N=862) 


11.25 


4.18 


16.82 


37.01 


30.74 
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SuiMIMARy Al\d CoNclusiONS 



This study followed on the heels of a statewide change in cutscores for placing students in 
English and Mathematics courses. The purpose of the study was to assess the impact of the 
changes in English scores on course grades in students’ first and subsequent English courses and 
to determine whether the change seemed warranted for the students whose test scores fell in the 
range where they were impacted by the change. 

Of the three tests used for placement, only one — the COMPASS — was truly developed to serve 
as a placement instrument. None of the three tests were developed with the aim of awarding 
academic credit. Thus, the relationship between performance in a particular course — as 
measured by course grades — and test scores could be weak. 

Indeed, based on grades in the first course, no good empirical evidence could be found to justify 
moving students out of ENGL 101 and into ENGL 090. Students who were targeted to move or 
were moved into ENGL 090 had similar grades to students who were not moved. Grade 
distributions also did not change. From this perspective, then, it appeared that students’ 
academic progress was unnecessarily slowed. However, grades in ENGL 101 were improved for 
students who began in ENGL 090 after the cutscore changes were implemented. Therefore, the 
movement of additional students into ENGL 090 appeared to be a mixed blessing: student 
progress was slowed, but ultimately many were better prepared for ENGL 101. 

In addition, no empirical evidence could be found for moving additional students into ENGL 
102. Grades were similar for those affected by the change and those who were not. Grade 
distributions did not change. Since the academic progress of about 120 students in three 
semesters was accelerated by the change, however, from a student perspective, the change was a 
good one. 

Students who were awarded credit for ENGL 1 02 under the new system typically had 
outperformed their classmates with lower placement scores. About 85% earned an “A” or a “B” 
in the course prior to the implementation of cutscores, indicating that students with this level of 
placement scores would handle ENGL 102 well. From this perspective, then, it would be good 
to award credit and accelerate their progress. However, this does not mean students who 
received credit had the skills which are addressed in ENGL 102. Since performance in the next 
course that required writing a research paper (a basic goal of ENGL 102) could not be assessed, 
the beneficial nature of this change remains uncertain. 




Research Report 2002-04 



iO 



9 



Appendix A 

Past and Current Cutscores for Placement in ENGL 090, 101, or 102 



Score Range: 


Before Fall 2000 


Fall 2000 and later 


ACT Writing 


14 or less 


Enroll in ENGL 090 


Enroll in ENGL 090 


15-17 


Enroll in ENGL 101 


Enroll in ENGL 090 


18-24 


Enroll in ENGL 101 


Enroll in ENGL 101 


25-30 


Enroll in ENGL 102 


Enroll in ENGL 102 


31-36 


Enroll in ENGL 102 


Get credit for ENGL 102 


SAT Verbal 


410 or less 


Take ENGL 090 


Take ENGL 090 


420-440 


Take ENGL 101 


Take ENGL 090 


450-560 


Take ENGL 101 


Take ENGL 101 


570-590 


Take ENGL 101 


Take ENGL 102 


600-690 


Take ENGL 102 


Take ENGL 102 


700-800 


Take ENGL 102 


Get credit for ENGL 102 


COMPASS Writing Test 


59 or less 


Enroll in ENGL 090 


Enroll in ENGL 090 


60-67 


Enroll in ENGL 101 


Enroll in ENGL 090 


68-94 


Enroll in ENGL 101 


Enroll in ENGL 101 


95-98 


Enroll in ENGL 101 


Enroll in ENGL 102 


99-100 


Enroll in ENGL 102 


Enroll in ENGL 102 



Note: Changes are shadec 
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